Reliability and Security of Deep Neural Networks

$\color{blue}{\text{Masoud Daneshtalab:}}$ Professor at Mälardalen University and TalTech

$\color{blue}{\text{Seyedhamidreza Mousavi:}}$ PhD student at Malardalan University

$\color{blue}{\text{Mohammad Hassan Ahmadilivani (Mojtaba):}}$ PhD student at TalTech University

$\color{blue}{\text{Email:}}$ masoud.daneshtalab@mdu.se, seyedhamidreza.mousavi@mdu.se, mohammad.ahmadilivani@taltech.ee

Reliability and security threats to machine learning-based systems

We are going to focus on two threats:

• Reliability issues (Hardware Faults) RReLU Framework
• Adversarial input perturbation ProARD Framework

First Part: Reliable ReLU Toolbox (RReLU) To Enhance Resilience of DNNs

What is a Deep Neural Network?

The application of DNNs: Object Detection

What is the Soft-errors problem?

❓ What is the issue with ReLU?

Soft-Error and it's effect on the DNNs Accelerators?

✅ What is the solution?

❓How can we find the bound value for each ReLU activation function?

❓If the output activation is higher than bound, what should we do?

❓IS there any real situation that changed the activation value a lot?

❓Do we need to find one bound value for each layer or neuron?

Data representation in Neural Network

Fixed-Point Number

Floating-Point Number

Activation Restriction Methods:

• Bounded ReLU activation functions:

• Algorithm for finding the bounds:
  • Ranger used the calibration method in a layer-wise way.
  • FT-ClipAct used a heuristic method based on fault-injection in a layer-wise manner.
  • FitAct leveraged an optimisation-based method in a neuron-wise mode.
  • ProAct used a Hybrid activation function (neuron-wise and layer-wise) and progressive training

ProAct provide an Open-Source Framework for all the Methods (RReLU)

It includes implementations of the following algorithms:

1. FitAct: Error Resilient Deep Neural Networks via Fine-Grained Post-Trainable Activation Functions

2. FT-ClipAct: Resilience Analysis of Deep Neural Networks and Improving their Fault Tolerance using Clipped Activation

3. Ranger: A Low-cost Fault Corrector for Deep Neural Networks through Range Restriction

4. ProAct: Progressive Training for Hybrid Clipped Activation Function to Enhance Resilience of DNNs

How do different methods work?

Ranger: Find the maximum value for a validation set and use that value as the bound in each ReLU.

FT-ClipAct:

FitAct:

ProAct:

Instructions to use RReLU framework

Install Packages and Clone the RReLU Github Repository

!pip install fxpmath
!git clone https://github.com/hamidmousavi0/reliable-relu-toolbox.git
%cd reliable-relu-toolbox/

Build Data Loader

First, we need to create the Dataset

from rrelu.setup import build_data_loader
data_loader_dict, n_classes = build_data_loader(dataset='cifar10',
                                    batch_size=128, image_size=32)
import matplotlib.pyplot as plt

# Get the first batch from the training data loader
images, labels = next(iter(data_loader_dict['train']))

# Get the class names for CIFAR-10
# This assumes CIFAR-10 is used as specified in the previous cell
class_names = ['airplane', 'automobile', 'bird',
               'cat', 'deer', 'dog', 'frog',
               'horse', 'ship', 'truck']


# plt.figure(figsize=(10,10))
# for i in range(25):
#     plt.subplot(5,5,i+1)
#     plt.xticks([])
#     plt.yticks([])
#     plt.grid(False)
#     # Transpose the image tensor to be in HxWxC format for plotting
#     plt.imshow(images[i].permute(1, 2, 0))
#     # The CIFAR10 labels are indices, so we use the class_names list
#     plt.xlabel(class_names[labels[i].item()])
# plt.show()

Build Model

• Our tool support pre-trained models on CIFAR-10, CIFAR-100, and ImageNet Dataset

• Cifar-10 and Cifar-100 supported models:

  • resnet20, resnet32, resnet44, resnet56
  • vgg11_bn, vgg13_bn, vgg16_bn, vgg19-bn
  • mobilenetv2_x0_5, mobilenetv2_x0_75
  • shufflenetv2_x1_5

• ImageNet supported Models:

  • All the models in the PyTorch-hub

from rrelu.setup import build_model
model = build_model(name='resnet20',dataset='cifar10', 
                    n_classes=n_classes)
print(model)

Evaluate the original Model with ReLU activation function

from metrics import eval_cpu
print(eval_cpu(model, data_loader_dict))

Convert Floating-Point weight values to the Fixed-Point

import torch
from fxpmath import Fxp
with torch.no_grad():
    for name, param in model.named_parameters():
        if param is not None:
            param.copy_(torch.tensor(Fxp(param.clone().cpu().numpy(),
                    True, n_word=32, n_frac=16, n_int=15).get_val(),
                                     dtype=torch.float32,device='cpu'))

Evaluate the Fixed-Point Model

print(eval_cpu(model, data_loader_dict))

Evaluating Reliability of the model

from metrics import eval_fault_cpu
print(eval_fault_cpu(model, data_loader_dict, 
                     1e-6, bitflip='fixed',iterations=5))

Build the model with Reliable ReLU

from rrelu.setup import replace_act
from metrics import eval_fault_cpu
model = replace_act(model, 'zero', 'ranger', data_loader_dict,
                    'layer', 'fixed',False ,'cifar10',is_root=True,)
print(eval_fault_cpu(model, data_loader_dict, 1e-6,
                     bitflip='fixed',iterations=5))

Results of using different methods:

GitHub project: reliable-relu-toolbox

Paper link: ProAct: Progressive Training for Hybrid Clipped Activation Function to Enhance Resilience of DNNs

Second Part: ProARD: Progressive Adversarial Robustness Distillation: Provide Wide Range of Robust Students

How about the perturbation in the input data? How can we defend against this type of attacks?

The state-of-the-art method is Adversarial Training.

The state-of-the-art methods are:

• TRADES: Theoretically Principled Trade-off between Robustness and Accuracy
• ARD: Adversarial Robustness Distillation: Use a Robust Teacher Network to train the student's networks. (What is the issue?)

Is it possible to train a single robust supernetwork and extract multiple robust subnetworks from it, each tailored to different resource-constrained devices—without requiring retraining?

How should we train this super-network?

• Randomly sample subnetworks from the supernetwork, train them independently, and return their updates to the supernetwork using weight sharing.
• Does it work?

First step: Making Super Dynamic Network:

If we trained the multiple subnetworks inside a supernetwork by randomly subsampling, what would be their accuracy and robustness?

We propose Progressive Adversarial Robustness Distillation (ProARD).

Enabling the efficient one-time training of a dynamic network that supports a diverse range of accurate and robust student networks without requiring retraining.

ProARD: Progressive Adversarial Robustness Distillation: Provide Wide Range of Robust Students

Project code:ProARD: Progressive Adversarial Robustness Distillation: Provide Wide Range of Robust Students

Results

$\color{blue}{\text{Refrences}}$

1. M. H. Ahmadilivani, M. Taheri, J. Raik, M. Daneshtalab, and M. Jenihhin,

“A systematic literature review on hardware reliability assessment methods for deep neural networks,” ACM Computing Surveys, vol. 56, no. 6, pp. 1–39, 2024. 2. L.-H. Hoang, M. A. Hanif, and M. Shafique, “Ft-clipact: Resilience analysis of deep neural networks and improving their fault tolerance using clipped activation,” in 2020 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2020, pp. 1241–1246.

3. B. Ghavami, M. Sadati, Z. Fang, and L. Shannon, “Fitact: Error resilient

deep neural networks via fine-grained post-trainable activation functions,” in 2022 Design, Automation & Test in Europe Conference & Exhibition (DATE). IEEE, 2022, pp. 1239–1244. 4. Mousavi, Seyedhamidreza, et al. "ProAct: Progressive Training for Hybrid Clipped Activation Function to Enhance Resilience of DNNs." arXiv preprint arXiv:2406.06313 (2024).

5. Mousavi, Seyedhamidreza, et al. "ProARD: Progressive Adversarial Robustness Distillation: Provide Wide Range of Robust Students " IJCNN (2025).
6. M. Goldblum, L. Fowl, S. Feizi, and T. Goldstein, “Adversarially

robust distillation,” in Proceedings of the AAAI conference on artificial intelligence, vol. 34, pp. 3996–4003, 2020.

Thank You